build: Demo of no-setup Makefile (standardized virtualenv handling) #347
Conversation
This allows a developer to run `make test` or other targets without having to manually set up a virtualenv or install requirements. The virtualenv is set up for them as a hidden subdirectory and requirements are automatically installed as needed. The goal here is larger: The new base.mk would be pulled in via git subtree from some centralized repository. Over time, this standardized base Makefile would grow and the Makefiles in regular repositories would shrink. By centralizing Makefile logic, we would gain the following: - Standardized names for make targets. Currently there is no universal way to install Python dependencies, to run all tests and quality checks, or to build docs. Developers have to learn what is used in each repo. We have standards for some of this, but they're not always followed. By composing Makefiles, we can ensure new repos always follow the standard, even as the standard changes. - Faster onboarding of developers who have not yet set up their own personalized virtualenv toolchains. Experienced Python developers already have these tools set up, but newer developers stumble on virtualenv creation. - A way to update repos to the latest Makefile improvements without having to manually update several hundreds of repos by hand. Automating the update of a git subtree reference still requires making and merging PRs but it does not require manual review of each PR, let alone manual PR creation.
| /.venv-3.8 | ||
|
|
There was a problem hiding this comment.
Author's note: In most repositories, tools like pytest are already configured such that they don't try to operate in the files inside the virtualenv since we call them with explicit paths, e.g. pylint edx_django_utils tests test_utils manage.py setup.py.
A few repos would need to be adjusted to follow this pattern. These repos are already a problem for people who use in-repo virtualenvs.
There would also need to be some adjustment for code searching. Developers who are used to using grep -nr ... would need to instead write grep -nr ... *. Same for ack, although adding --ignore-dir=match:/^.?venv-[0-9.]+$/ to one's ~/.ackrc will prevent searching any such virtualenv. Those who use ripgrep would already be covered, as ripgrep ignores hidden dirs by default.
|
|
I think the main use for a shared Docker image would be getting the right version of Python, yeah. That might be a separate PR, but it looks like we could add something like this to the Makefile: with an image based on a Dockerfile that looks something like this: Seems like it works based on some local testing -- |
|
As a branch: timmc/demo-venv...timmc/demo-make-container |
This allows a developer to run
make testor other targets without having to manually set up a virtualenv or install requirements. The virtualenv is set up for them as a hidden subdirectory and requirements are automatically installed as needed.The goal here is larger: The new base.mk would be pulled in via git subtree from some centralized repository. Over time, this standardized base Makefile would grow and the Makefiles in regular repositories would shrink. By centralizing Makefile logic, we would gain the following: